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While  many  studies  have  compared  passive  and  active  instructional  methods, 
none  provides  statistical  evidence  that  one  method  is  clearly  superior.  When 
the  subject  matter  to  be  taught  is  technical  in  nature,  however,  the  experiential 
method  has  been  shown  to  be  more  effective  in  terms  of  both  student 
reactions  and  learning. 


The  environment  faced  by  most  or 
ganizations  today  is  characterized  by 
increasing  dependence  on  technol¬ 
ogy,  whether  that  technology  is  a  product 
of  the  organization  or  the  tool  it  uses  to 
compete  in  the  marketplace.  In  addition 
to  the  emergence  of  technology  as  a  domi¬ 
nant  consideration  in  the  strategy  of 
organizations,  the  pace  of  technological 
change  has  accelerated  to  the  point  that 
technical  obsolescence  is  a  concern  that 
affects  both  products  and  people.  In  1994 
the  Motorola  Corporation  estimated  that 


the  knowledge  of  the  average  engineer 
becomes  obsolete  in  2  to  5  years 
(Motorola,  1995). 

Furthermore,  competition  has  become 
global  in  nature,  increasing  the  pres¬ 
sures  on  organizations  to  be  both  tech¬ 
nically  agile  and  to  be  economically 
efficient  producers.  In  this  environment 
of  complex  technologies,  rapid  techni¬ 
cal  obsolescence,  and  global  competi¬ 
tion,  organizations  have  increasingly 
accepted  the  idea  that  education  and 
training  are  key  to  the  ability  of  the 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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organization  to  compete  in  the  global 
economic  environment. 

Training  is  a  major  commitment  for 
technically  focused  firms  today,  and  it  has 
become  a  big  business.  Motorola,  for 
example,  affords  each  of  its  132,000 
employees  one  week  of  training  each  year 
(Motorola,  1995),  and,  according  to  one 
estimate  (Dipoye,  Smith,  and  Howell, 
1994),  U.S.  corporations  having  100  or 
more  employees  spent  $43.2  billion  for 
training  during  1991.  With  this  level  of 
commitment  in  mind,  it  becomes  particu¬ 
larly  important  that  training  be  conducted 
as  effectively  as  possible,  and  that  the 
instructional  methods  used  be  those  that 
contribute  most  to  improving  the  job 
performance  of  the  student. 

This  article  describes  a  study  that  made 
a  side-by-side  comparison  of  instructional 
methods  used  to  teach  two  large  groups 

of  students  at¬ 
tending  the 
Advanced  Pro¬ 
gram  Manage¬ 
ment  Course 
(APMC)  at  the 
Defense  Sys¬ 
tems  Manage¬ 
ment  College 
(DSMC).  One  group  was  taught  the 
systems  engineering  management  por¬ 
tion  of  the  course  using  lecture  and  dis¬ 
cussion  as  the  primary  teaching  method, 
while  the  second  group  was  taught 
experientially  using  a  hands-on  design 
project.  The  learning  objectives  of  the  two 
courses  were  identical  and  the  student 
groups  evaluated  were  very  homogeneous. 
This  appeared  to  be  an  excellent  opportu¬ 
nity  to  conduct  a  controlled  study  in  order 
to  investigate  the  impact  of  teaching 
methods  on  learning  outcomes.  Given  the 


substantial  commitment  made  by  business 
and  government  to  education  and  train¬ 
ing,  the  results  of  this  investigation  could 
be  important  in  determining  the  nature  of 
training  for  students  in  future  courses. 

Related  Literature 


Perhaps  the  best  known  and  most  com¬ 
prehensive  approach  to  the  assessment  of 
training  was  developed  by  Kirkpatrick 
(1977).  His  model  comprised  four  discrete 
and  progressive  levels  of  evaluation: 
student  reactions,  learning  achieved, 
transfer  of  changed  behaviors  to  the 
workplace,  and  results  achieved  in  the 
workplace.  Most  evaluation  does  not  go 
beyond  assessment  of  student  reactions 
(Dipoye,  Smith,  and  Howell,  1994)  and 
appears  to  be  based  on  the  assumption 
that,  if  the  student  leaves  the  course  with 
a  positive  attitude  regarding  the  training, 
then  there  will  be  positive  results  in  other 
measures  of  effectiveness. 

Research  indicates  that  this  assumption 
is  likely  to  be  a  poor  one.  In  a  study  evalu¬ 
ating  the  relationships  among  the  differ¬ 
ent  levels  in  the  Kirkpatrick  model,  Alliger 
and  Janak  (1989)  found  no  significant 
relationship  between  student  reactions  to 
training  and  the  higher  levels — learning, 
transfer  to  the  workplace,  or  results 
achieved.  This  makes  intuitive  sense;  the 
course  rated  favorably  by  students  may 
not  be  the  one  that  provides  useful  learn¬ 
ing.  On  the  other  hand,  Alliger  and  Janak 
found  a  positive  relationship  among  the 
higher  levels,  indicating  that  learning, 
once  achieved,  will  likely  result  in  the 
transfer  of  lessons  learned  to  the  work¬ 
place,  and  subsequently  to  improved 
results  in  the  workplace. 


"Training  is  a 
major  commitment 
for  technically 
focused  firms  today, 
and  it  has  become 
a  big  business." 
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The  fact  that  there  is  positive  linkage 
among  the  higher  three  levels,  but  no 
relationship  between  student  reaction  and 
learning,  suggests  that  learning  is  the  criti¬ 
cal  indicator  of  course  effectiveness;  learn¬ 
ing  is  the  linchpin.  If  the  student  learns, 
the  course  is  likely  to  result  in  a  positive 
change  at  all  levels  of  effectiveness. 
Others,  such  as  Landy  (1987)  and  Maier 
(1973),  have  observed  that,  while  the 
prevailing  assumption  has  been  that 
attitudes  influence  behavior,  for  this  to  be 
true  requires  a  confluence  of  other  factors, 
such  as  experience  and  motivation.  Evalu¬ 
ation  that  stops  at  the  assessment  of 
student  reaction  too  often  provides  little 
in  the  way  of  useful  information  regard¬ 
ing  the  probability  that  the  training  results 
in  learning  or  that  the  training  will  carry 
over  into  the  workplace. 

Adult  learning  theory,  as  espoused  by 
Knowles  (1980)  and  others,  implies  that 
adults  will  learn  and  retain  more  when 
they  take  an  active  role,  participate  more, 
and  use  multiple  senses  in  the  learning 
process.  A  number  of  research  efforts  have 
been  conducted  with  the  objective  of 
evaluating  the  extent  to  which  teaching 
methods  influence  learning,  but  few 
statistically  significant  conclusions  have 
been  recorded.  According  to  Rachal 
(1994),  “...advocacy  of  andragogy  as  a 
superior  strategy  for  facilitating  adult 
learning  does  not  seem  to  be  borne  out 
by  the  existing  empirical  studies...  .” 

Table  1  summarizes  a  number  of  stud¬ 
ies  that  have  compared  teaching  methods. 
All  compared  various  participative  and 
experiential  methods  with  a  control  group 
taught  using  more  passive  methods.  Sig¬ 
nificantly,  none  established  a  statistical 
difference  in  the  learning  achieved  using 
one  method  in  preference  to  the  other. 


Campbell  noted  in  his  later  work  (1988) 
that  analyses  performed  in  this  topic  area 
are  increasingly 
well  structured 
and  rigorous; 
however,  even 
in  more  recent 
studies,  there  is 
little  statistical 
evidence  that  ex¬ 
periential  meth¬ 
ods  are  superior  to  more  passive  methods. 

In  spite  of  this  consistent  pattern  of  fail¬ 
ure  to  find  differences  in  the  results 
achieved  between  instructional  methods 
in  various  types  of  training  courses,  the 
present  study  was  structured  specifically 
to  compare  the  effectiveness  of  alterna¬ 
tive  methods.  Many  of  the  studies 
reviewed  seemed  to  have  one  or  more 
problems: 

•  In  some  cases  the  differentiation 
between  “participative”  and  “nonpart- 
icipative”  conditions  was  nebulous. 

•  In  many  cases  the  samples  were  quite 
small,  so  the  power  to  discriminate  was 
lessened. 

•  Where  the  samples  were  reasonably 
large,  the  research  was  conducted  in 
environments  (universities,  corpora¬ 
tions)  that  are  apt  to  be  subject  to  a 
great  deal  of  outside  interference 
(Walleri  and  Japely,  1986). 

The  numbers  of  subjects  available  at 
DSMC,  the  homogeneity  of  the  groups 
who  attend  courses  (Table  2),  and  the  fact 
that  students  are  largely  isolated  from  their 
work  and  outside  interference  suggested 
that  the  environment  would  be  ideal  for  a 


"If  the  student 
learns,  the  course 
is  likely  to  result 
in  a  positive  change 
at  all  levels  of 
effectiveness.” 
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Table  1.  Related  Studies  of  Comparative  Teaching  Methods 


Author 

Group 

N 

Methods 

Compared 

Instruments 

Analytic 

Methods 

Results 

(Learning) 

Bretz  and 

Thompsett 

(1992) 

Kodak  Corp. 
MRP  Training 

180 

Lecture  vs. 
integration 

Survey, 

pretest, 

learning 

ANOVA 

Correlation 

posttest 

No 

difference 

P=.01 

Carr 

(1982) 

University 

(Economics) 

26 

Lecture  vs. 
case  study 

Pretest, 

posttest 

T-test, 

ANOVA 

No 

difference 

P  =  .05 

Carter 

(1995) 

University 
(physical  training) 

36 

Lecture  vs. 
case  study 

Survey, 

posttest 

(only) 

MANOVA 

No 

difference 

P=.10 

James 

(1991) 

Adult 

(education) 

31 

Lecture  vs. 
case  study 

Pretest, 

posttest 

ANOVA 

T-test 

Regression 

No 

difference 
P=  .05 

Merrill 

(1995) 

Adult  medical 
(cardiac) 

37 

Lecture  vs. 
self-study 

Pretest, 

posttest 

ANCOVA 

No 

difference 
P=  .05 

Thoms 
and  Klein 
(1994) 

Adult 

(hospital 

management) 

64 

Nonparticipation 

vs. 

participation 

Survey, 

multiple 

tests 

ANOVA 

Correlation 

Chi-Square 

No 

difference 
r=  .16 

Ward 

(1993) 

Navy 

(medical) 

300 

Nonparticipation 

vs. 

participation 

Survey, 

posttest 

(only) 

T-test, 

ANOVA 

Learning 
not  assessed 

Welch 

(1990) 

University 

students 

(business) 

181 

Lecture  vs. 
active  methods 

Pretest, 

posttest 

ANOVA 

No 

difference 
P=  .05 

White 

(1995) 

University 

students 

(tech) 

112 

Lecture  vs. 
computer-aided 
training 

Survey, 

pretest, 

posttest 

ANOVA 

No 

difference 

P=.10 

comparison  of  teaching  methods  in 
conditions  where  those  methods  were 
likely  to  be  the  primary  determinant  of 
differences  observed  in  performance.  This 
is  an  issue  of  particular-  interest  since  many 
organizations  are  moving  rapidly  to 
implement  curricula  that  are  experiential 
and  hands-on  (Parkinson,  1994;  Raelin 
and  LeBien,  1993)  rather  than  lecture- 
based  and  theoretical,  in  spite  of  the  lack 
of  empirical  research  findings  that  support 
that  trend. 


Mcthod 


SlIBJ  ECTS  AND  M ETHODS  COMPARED 

The  APMC  is  a  training  course  for 
Department  of  Defense  (DoD)  program 
managers,  conducted  at  DSMC.  Those 
attending  the  course  are  selected  from  that 
portion  of  the  DoD  acquisition  workforce 
who  either  hold  senior  management 
positions  (program  managers,  deputy  pro¬ 
gram  managers,  functional  managers,  and 
division  heads)  in  program  offices,  or  who 
are  being  prepared  for  such  positions. 
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While  they  take  the  course,  they  are 
relieved  from  their  assigned  jobs,  are 
relocated  to  DSMC,  and  are  expected  to 
attend  classes  daily  in  various  topics 
related  to  project  management. 

APMC  95-1  was  taught  the  engineer¬ 
ing  management  portion  of  the  course 
using  lecture-discussion  methods  with 
limited  exercises.  APMC  96-1  entered  a 
year  later,  but,  rather  than  lecture,  this 
group  was  taught  by  integrating  the 
instruction  with  a  design  project.  The 
students  in  APMC  96- 1  were  required  to 


plan,  design,  build,  and  then  test  a  vehicle 
based  on  a  set  of  performance 
requirements  issued  at  the  beginning  of 
the  course.  All  engineering  management 
instruction  was  woven  into,  and  was 
related  to,  the  design  project.  APMC  95-1 
was  the  control  group,  and  APMC  96-1 
was  the  treatment  group  for  purposes  of 
this  study. 

Each  offering  of  the  APMC  included 
approximately  420  students — a  represen¬ 
tative  mix  of  military  and  civilian  employ¬ 
ees  in  DoD  research  and  development 


Table  2.  Demographics  of  Control  and  Treatment  Groups 


Attribute 

APMC  95-1 

APMC  96-1 

Affiliation 

Air  Force 

29.7% 

30.5% 

Army 

28.9% 

25.0% 

Navy/Marines 

28.6% 

32.9% 

Other 

12.9% 

1 1 .6% 

Highest  education  level 

Masters/Ph.D. 

69.4% 

71.4% 

Bachelors/other 

30.6% 

28.6% 

Education  type 

Technical 

48.6% 

47.1% 

Management/other 

51.4% 

52.9% 

Military 

55.3% 

50.7% 

Civilian 

44.7% 

49.3% 

Female  attendees 

13.6% 

16.9% 

Acquisition  experience  (years) 

9.5 

10.8 

DoD  experience  (years) 

15.5 

17.4 

Sample  total 

360a 

420 

a  Two  sections  (60  students)  were  excluded  from  the  APMC  95-1  control  group,  because 
they  were  used  to  pilot  the  techniques  that  were  later  used  in  teaching  the  APMC  96-1  course. 
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activities.  In  addition,  a  few  employees 
of  defense  industry  corporations  typically 
attend  each  class  offering.  Table  2  com¬ 
pares  the  demographics  of  the  two  groups 
involved  in  this  study.  The  homogeneity 
of  the  students  who  attend  these  courses 
is  remarkable — they  are  of  similar  ages 
(typically  35  to  40  years  old);  they  tend 
to  have  similar  educational  and  experience 
backgrounds,  and  they  are  similar  in 
civilian/military  mix  as  well  as  in  service 
affiliation. 

Each  class  was  divided  into  14  sections 
of  30  students.  Section  assignment  was  a 
stratified  random  process;  subgroups  were 
established  proportionally  by  service 
affiliation  (Army,  Navy,  Air  Force),  then 
assignment  to  sections  was  random.  The 
result  was  a  distribution  that  was  pro¬ 
portionally  representative  of  the  entire 
class  in  terms  of  service  or  industry 
affiliation,  and  which  was  random  in 
terms  of  assignment  of  individuals  to 
sections. 

Sections  received  the  same  program  of 
instruction  based  on  an  established 


curriculum  with  learning  objectives  that 
were  identical  both  among  sections  and 
between  the  two  class  groups.  Each  sec¬ 
tion  was  taught  by  an  assigned  instructor. 
The  12  sections  in  the  control  group  were 
taught  by  eight  instructors  (four  of  whom 
taught  two  sections);  the  14  sections  of 
the  treatment  group  were  taught  by  12 
instructors  (5  of  whom  had  taught  the 
control  group).  Neither  instructors  nor 
students  were  aware  that  student  data 
would  be  analyzed  for  the  puiposes  of  this 
study.  A  concern  that  the  control  group 
(95-1)  might  have  treated  the  course  with 
less  seriousness  (since  this  was  the  last 
offering  of  the  format  in  use  at  the  time) 
were  resolved  by  comparing  the  end  of 
course  comprehensive  exam  scores  of 
APMC  95-1  with  previous  classes.  Their 
scores  were  essentially  equal  to  those  of 
their  predecessors,  indicating  no  lessening 
of  effort  or  learning  on  their  paid. 

Instruments 

This  study  evaluated  the  effectiveness 
of  the  training  offered  in  terms  of  the  first 


1 .  Are  you  employed  as  a 

a.  Government  civilian 

b.  Military 

c.  Contractor 

d.  None  of  the  above 

2.  Highest  level  of  formal  education  achieved 

a.  Bachelor’s  degree 

b.  Master’s  degree 

c.  Doctoral/Ph.D. 

d.  Other  (specify) _ 


Figure  1.  Sample  Items  from  Pretest  Demographic  Survey 
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two  levels  of  the  Kirkpatrick  model,  and 
consequently  measurement  of  student 
reactions  and  learning  were  key.  The 
collection  of  data  was  handled  through 
administration  of  pretest  and  posttest 
instruments. 

Pretest.  The  pretest  included  a  test  of 
knowledge  in  the  engineering  manage¬ 
ment  domain  and  a  survey  of  demographic 
factors.  The  domain  knowledge  test  is 
discussed  in  more  detail  below.  The 
demographic  questionnaire  surveyed 
employment  and  educational  background, 
experience,  and  previous  training  prior  to 
exposure  to  the  APMC  course.  An 
example  of  two  items  on  the  demographic 
survey  is  shown  in  Figure  1 . 

Posttest.  The  posttest  consisted  of  a 
questionnaire  that  measured  student 
reactions  to  the  course  and  a  second  test 
of  knowledge  in  engineering  manage¬ 
ment.  Both  were  administered  at  the  final 
training  session.  The  questionnaire 
included  five  questions  and  also  provided 
for  subjective  student  comments.  Three 
of  the  questions  addressed  the  extent  to 
which  students  found  the  course  of 
instruction  informative,  enjoyable,  and 
characterized  by  reasonable  workloads; 
these  were  the  questions  viewed  to  best 


reflect  student  reactions  to  the  course,  as 
described  by  Kirkpatrick.  The  remaining 
items  dealt  with  preferences  for  hands-on 
versus  lecture  and  the  future  usefulness 
of  the  training  in  the  work  environment. 
Responses  were  given  in  numerical  form 
using  a  seven-point  Likert  scale.  Figure  2 
shows  a  sample  question. 

Domain  knowledge  test.  The  domain 
knowledge  test  was  the  means  for  mea¬ 
suring  learning.  A  panel  of  subject  matter 
experts  in  engineering  management  from 
DSMC  evaluated  all  questions  proposed 
for  content  validity.  This  panel  consisted 
of  six  experienced  systems  engineering 
managers  who  were  teaching  in  the 
APMC,  all  of  whom  were  Level  III  mem¬ 
bers  of  the  Defense  Acquisition  Work 
Force.  Questions  were  formally  scored  on 
the  extent  to  which  they  satisfactorily 
addressed  the  learning  objectives  with 
which  they  were  associated.  These  learn¬ 
ing  objectives  were  derived  from  the  set 
of  competencies  required  of  the  systems 
engineering  management  subcourse  in  the 
APMC. 

The  internal  reliability  of  the  instru¬ 
ments  was  evaluated  over  a  period  of 
several  months  by  testing  them  on  several 
groups  of  students  who  were  attending 


1  found  the  systems  engineering  course  material  informative. 

Strongly 

Strongly 

disagree 

1  1  1 

agree 

i  i  i 

1  1  1 

1 

4 

i  i  i 

7 

Figure  2.  Sample  Item  from  Posttest  Student  Reaction  Questionnaire 
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DSMC  short  courses  in  systems  engineer¬ 
ing  management.  In  terms  of  experience 
levels  and  other  key  variables  such  as 
educational  backgrounds,  the  attendees  at 
Level  III  short  courses  are  comparable  to 
the  attendees  at  APMC.  The  major  differ¬ 
ence  is  that  the  attendees  at  the  special¬ 
ized  short  courses  in  engineering  manage¬ 
ment  tend  to  be  predominantly  from  the 
technical  disciplines  and  to  hold  techni¬ 
cal  management  positions,  while  the 
APMC  includes  a  more  balanced  mix  of 
managers  from  various  functional  areas. 

Tests  were  administered  using  the  split- 
half  procedure.  As  experience  was  gained 
through  repeated  testing  in  succeeding 
courses,  adjustments  were  made  in  the 
content  and  mix  of  the  tests  based  on  item 
analyses.  The  reliability  was  finally  dem¬ 
onstrated  to  be  0.930.  As  an  additional 
check,  the  entire  question  set  was  also 


administered  as  a  single  exam,  and  the 
reliability  was  calculated  using  the  Kuder- 
Richardson  (KR  20)  formula.  The  results 
of  that  calculation  indicated  that  the  mean 
reliability  of  all  split  half  combinations 
is  0.83,  still  well  within  the  bounds  nor¬ 
mal  to  skills  and  competency  based  tests 
in  use. 

The  full  question  set  was  divided  into 
halves  with  demonstrated  reliability  0.93. 
In  their  final  form  the  two  tests  (pretest 
and  posttest)  were  each  15  questions  that 
tested  student  knowledge  at  the  applica¬ 
tion  level  and  which  had  been  demon¬ 
strated  to  do  so  equally.  An  example  from 
one  of  those  tests  is  shown  in  Figure  3. 

Procedure.  The  pretest  instrument  was 
administered  to  both  the  control  and  treat¬ 
ment  groups  during  their  first  lesson  in 
the  systems  engineering  subcourse.  The 
posttest  was  administered  to  both  groups 


As  your  program  approaches  CDR,  your  systems  engineer  informs  you 
that,  in  his/her  estimation,  the  design  is  about  60%  complete.  Assuming 
that  his/her  estimate  is  correct,  you  should  (choose  one): 

a.  Continue  as  planned  and  hold  the  CDR  as  scheduled.  In  today’s 
environment,  design  maturity  is  not  an  issue  for  government 
program  managers. 

b.  Hold  the  CDR  as  scheduled.  There  are  minimal  risks  associated 
with  early  CDRs,  since  designs  continue  to  mature  until  well  after 
CDR  is  completed. 

c.  Delay  holding  the  CDR  until  the  design  is  substantially  complete. 

A  design  should  be  85-90%  complete  before  the  CDR  is  con¬ 
ducted. 

d.  Delay  the  CDR  until  the  contractor  completes  and  delivers  the 
system  specification  for  government  review. 


Figure  3.  Sample  Domain  Knowledge  Test  Item 
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Table  3.  Two-Sample  t  Test  for  Pretest3 


Class 

N 

Mean 

SD 

t 

APMC  95-1 

359 

57.2 

15.9 

1.02b 

APMC  96-1 

407 

58.4 

16.2 

aHo:  APMC  95-1  =  APMC  96-1 . 
b  P°=  .30. 

at  the  end  of  that  course.  Taken  together, 
the  two  provided  the  means  to  compare 
knowledge  levels  between  and  within 
groups,  both  before  and  after  exposure  to 
the  engineering  management  course  of 
training.  They  also  provided  the  means 
to  relate  knowledge  of  engineering  man¬ 
agement  concepts  and  principles  to 
demographic  factors,  and,  finally,  they 
measured  student  reactions  to  the  train¬ 
ing  received.  The  administration  of  these 
two  instruments  completed  the  data 
collection  for  the  conclusions  drawn  here. 

This  was  a  pretest-posttest  control 
group  experimental  study.  The  span  of 
time  between  the  pretest  and  posttest 
administration  was  10  weeks.  Pretest- 
posttest  interaction  was  controlled  in  this 
case  by  the  timing  of  the  two  tests  and  by 
using  different  questions  in  the  pretest  and 
posttest.  The  differences  in  the  pretest  and 
posttest,  in  combination  with  a  span  of 
more  than  two  months  between  measure¬ 
ments,  was  considered  adequate  to  guard 
against  pretest-posttest  interaction. 

Results  of  the  Analyses 


Initial  Knowledge  Levels 

While  the  data  in  Table  2  indicate 
strong  similarity  between  the  groups,  it 


was  important  to  determine  whether  or  not 
the  two  sample  groups  were  equal  with 
respect  to  the  level  of  knowledge  upon 
entry  to  the  course.  The  results  (Table  3) 
indicate  that  the  initial  knowledge  level, 
as  shown  by  mean  pretest  scores  of  the 
two  groups,  was  equal  ( P  =1.05).  Initial 
equality  between  groups  made  more  sup¬ 
portable  a  conclusion  that  the  differences 
observed  later  could  be  attributed  to 
exposure  to  the  training,  rather  than  to 
differences  that  may  have  existed  at  the 
outset  of  the  experiment. 

The  analysis  then  turned  to  investiga¬ 
tion  of  hypotheses  that  were  structured  to 
parallel  the  first  two  levels  of  the 
Kirkpatrick  model  (1977).  These  were  that 
students  would  react  more  favorably  to  a 
curriculum  taught  using  experiential 
methods,  and  that  students  would  learn 
more  from  curriculum  taught  using 
experiential  methods. 

Student  Reactions 

Student  reaction  data  were  collected 
using  the  posttest  questionnaire  described 
earlier.  The  three  variables  used  as  indi¬ 
cators  of  student  reactions  and  associated 
statements  were: 

•  Informed:  I  found  the  engineering 

management  course  informative. 
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•  Enjoyed:  I  enjoyed  the  engineering 
management  course. 

•  Workload:  The  course  covered  too 
much  material  for  the  time  allotted. 

Analytic  results  indicate  that  APMC 
96-1.  the  class  taught  using  experiential 
methods,  believed  that  the  class  was  more 
informative,  enjoyed  the  course  more,  and 
perceived  the  workload  to  be  less  oppres¬ 
sive  (Table  4).  This  last  finding  is  inter¬ 
esting  in  that  the  requirement  to  design, 
fabricate,  and  test  vehicles  appeared  to 
have  represented  a  considerably  increased 
workload  over  that  associated  with  the 
course  taught  using  lecture-discussion 
methods,  yet  the  students  indicated  that 
they  perceived  the  workload  to  be  less. 

Narrative  reactions  were  also  solicited 
from  the  students  of  the  two  courses.  The 


results,  while  more  subjective,  corrobo¬ 
rated  the  analytic  results  of  the  t  tests  in 
that  the  comments  of  APMC  95-1 
indicated  less  satisfaction  than  did  APMC 
96-1. 

Learning 

A  number  of  studies  (e.g.,  Bretz  and 
Thompsett,  1992)  have  concluded  that 
exposure  to  experiential  instruction  will 
result  in  student  reactions  that  are  more 
positive  than  those  elicited  from  classes 
exposed  to  the  more  traditional  lecture  and 
discussion  approach  to  teaching.  The 
results  reported  above  confirm  and 
strengthen  these  findings. 

Theory  leads  one  to  expect  that  the  ex¬ 
periential  methods  would  produce  supe¬ 
rior  results,  not  only  in  student  reactions, 
but  also  in  learning.  To  measure  learning, 
this  study  used  the  combination  of  the 


Table  4.  Two  Sample  t  Test  for  Student  Reactions 


Variable 

N 

Mean 

(Scale  of  1  to  7) 

SD 

t 

Informed 

APMC  95-1 

321 

4.67 

1.30 

-4.1 2a 

APMC  96-1 

393 

5.12 

1.57 

Enjoyed 

APMC  95-1 

322 

4.51 

1.41 

—5 . 0 1 a 

APMC  96-1 

393 

5.09 

1.64 

Workload 

APMC  95-1 

322 

3.41 

1.48 

4.78a 

APMC  96-1 

392 

2.88 

1.44 

aP=  .01. 
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Table  5.  Student  Comments  on  Engineering  Management  Course 


Category 

APMC  95-1  (%) 

APMC  96-1  (%) 

Satisfied  (commented  positively) 

10 

14 

Dissatisfied  (commented  negatively) 

17 

15 

Satisfied/neutral  (commented  generally, 
but  neither  favorably  nor  unfavorably) 

13 

17 

Neutral  (did  not  comment) 

60 

54 

pretest  and  posttest  to  measure  knowledge 
levels  at  the  beginning  of  training  and  at 
the  completion.  Analysis  of  covariance 
was  used  to  adjust  observed  posttest 
scores,  controlling  for  differences  ob¬ 
served  in  pretest  results.  We  used  t  tests 
to  measure  both  the  extent  to  which  each 
instructional  method  resulted  in  positive 
learning  (pretest  vs.  posttest)  and  to  mea¬ 
sure  the  extent  to  which  one  method  pro¬ 
duced  more  learning  than  the  other 
(posttest  vs.  posttest).  Table  6  gives  the 
results  of  those  tests  (pretest  scores  are 
repeated  for  convenience). 

These  tests  indicate  that  both  lecture 
and  experiential  methods  produced 
positive  learning.  The  tests  furthermore 
indicate  that  the  experiential  method 
produced  a  higher  level  of  knowledge  as 
indicated  by  posttest  scores.  This  statisti¬ 
cally  supports  the  theoretical  hypothesis 


that  experiential  training  produces  an 
improved  level  of  learning;  however,  the 
differences  measured  were  smaller  than 
might  have  been  expected  or  hoped  for. 
More  will  be  said  on  this  topic  later. 

Relationship  between  Student 
Reactions  and  Learning 

Alliger  and  Janak  (1989)  found  no 
significant  relationship  between  student 
reactions  and  learning.  This  study  pre¬ 
sented  an  opportunity  to  evaluate  whether 
or  not  that  same  lack  of  relationship  was 
in  evidence  in  this  study,  as  well.  Since 
the  demographics  (Table  2)  and  initial 
analysis  of  pretest  scores  indicated  that 
the  two  groups  involved  could  reasonably 
be  considered  to  represent  a  single  under¬ 
lying  population,  the  two  were  combined 
into  a  single  large  group  for  purposes  of 
this  analysis. 


Table  6.  Domain  Knowledge  Test  Scores 


Group 

Pretest 

Mean 

Posttest 

Mean 

Posttest 
Mean  (Adj.) 

Posttest 

SD 

fTest 

(Pre-Post) 

fTest 

(Post-Post) 

APMC  95-1 

57.2 

73.2 

73.4 

16.6 

-12.673 

-3.93a 

APMC  96-1 

58.4 

77.8 

77.7 

14.1 

-17.893 

» P=  .01. 
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Table  7. 

Multiple  Regression— Student  Reactions  and  Pretest  on  Posttest  Score3 


Predictors 

Dependent  Variable,  Posttest  Score 

B 

SD 

Seq.  SS 

t 

P 

Constant 

64.332 

3.367 

19.1 

0.000 

Informed 

-0.594 

0.849 

188.6 

-0.70 

0.484 

Enjoyed 

0.360 

0.800 

5,13.7 

0.45 

0.653 

Workload 

-1.754 

0.392 

6,596.3 

-4.48 

0.000 

Pretest 

0.301 

0.037 

16,374.1 

8.34 

0.001 

aR-Sq  (Adj.)  =  1 1 .9%. 

Using  the  variables  “enjoyed,”  “in¬ 
formed,”  and  “workload”  (described 
earlier)  as  the  indicators  of  student 
reactions,  we  performed  a  multiple  regres¬ 
sion  on  posttest  score.  While  “workload” 
was  significant  ( P  =  .05),  these  variables 
together  explained  little  of  the  total 
variance  in  posttest  scores,  leading  to  the 
conclusion  that  student  reactions  are 
essentially  unrelated  to  learning,  and 
thus  supporting  the  Alliger  and  Janak 
observation. 

As  a  further  check,  pretest  scores  were 
then  added  as  an  independent  variable. 
Table  7  shows  the  result  of  the  regression 
analysis.  While  the  addition  of  pretest 
scores  improved  the  model  somewhat,  the 
coefficient  of  determination  remained 
quite  low,  indicating  that  the  outcome  of 
the  course  of  instruction  was  primarily 
explained  by  factors  other  than  either  pre¬ 
testing  or  student  reactions.  There  was  a 
moderate  correlation  (r  -  .33)  between 
pretest  and  posttest  scores,  i.e.,  those  who 
score  well  on  the  pretest  tend  to  score  well 
on  the  posttest  and  vice  versa. 


Other  Factors  that  Influence 
Initial  Levels  of  Knowledge 

The  pretest  instrument  included  a  sur¬ 
vey  that  requested  information  regarding 
the  student’s  job  status,  educational  level, 
educational  specialty,  previous  work  ex¬ 
perience,  and  professional  training.  The 
final  part  of  the  study  involved  analysis 
of  the  extent  to  which  pretest  score  was 
influenced  by  these  categorical  factors. 
Pretest  scores  were  used  as  an  indicator 
of  typical  knowledge  levels  among  the 
acquisition  workforce  population  at  large. 
The  data  were  grouped  to  enable  analysis 
of  factors,  each  with  multiple  states  as 
shown  in  Table  8.  Factorial  AN OVA  was 
used  for  the  analysis. 

Education  type,  years  of  program  man¬ 
agement  experience,  and  the  nature  of 
training  courses  taken  previously  were 
most  significant.  Somewhat  surprising 
was  the  finding  that  the  nature  of  current 
job  held  (technical  or  nontechnical)  and 
the  interval  since  attendance  at  training 
courses  did  not  produce  significant 
differences. 
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Table  8.  Effect  of  Categorical  Factors  on  Pretest  Performance 


Factor 

States  Addressed 

Adj.  SS 

F 

P 

Employment  status 

Military  or  civilian 

121.1 

0.52 

0.470 

Education  level 

Graduate  versus 
other 

968.3 

4.18 

0.041 

Type  of  education 

Technical  versus 
nontechnical 

4,313.9 

18.61 

0.000 

Current  job 

Technical  versus 
nontechnical 

114.6 

0.49 

0.482 

Management 

experience 

Years  of 
experience 

2,581.9 

11.14 

0.001 

Gender 

Male  or  female 

672.1 

2.90 

0.089 

Previous  training 
courses 

None,  general, 
or  functionally 
specific 

3,338.5 

14.41 

0.000 

Interval 

Years  since 
last  training 

14.9 

0.06 

0.800 

Error 

df=  693 

160,602.5 

Since  the  type  of  educational  back¬ 
ground  (technical  or  nontechnical)  that  the 
individual  possessed  produced  very 
significant  differences  in  test  scores,  it 
might  have  been  expected  that  the  type  of 
job  currently  held  would  produce  similar 
differences.  Formal  technical  education 
appears  to  convey  a  broader,  more 
integrative  level  of  knowledge  that 
contributes  to  the  individual’s  ability  to 
organize  and  control  resources  to  accom¬ 
plish  a  technical  objective  (engineering 
management),  while  holding  a  technical 
job  did  not.  The  nature  of  technical  work 
is  often  highly  specialized  and  relatively 
narrowly  focused,  which  may  explain  the 
fact  that  those  with  technical  jobs  did  not 
score  significantly  better  on  the  pretest. 

The  nature  of  previous  training  courses 
attended  deserves  comment,  also.  The 


prerequisite  training  courses  attended  by 
students  at  APMC  tend  to  fall  into  two 
general  categories:  those  that  address  pro¬ 
gram  management  in  general  by  exposing 
the  student  to  relatively  short  subcourses 
in  a  variety  of  disciplines  (of  which  engi¬ 
neering  management  is  one),  and  those 
that  are  focused  on  a  specific  management 
discipline  (such  as  engineering  management 
or  contract  management). 

This  study  found  that  students  exposed 
to  multidisciplinary  general  management 
courses  did  not  perform  significantly 
differently  on  the  engineering  manage¬ 
ment  pretest  from  those  who  had  not 
attended  previous  training.  On  the  other 
hand,  those  who  had  attended  training  that 
was  functionally  specialized  to  teach 
engineering  management  performed 
significantly  better  on  the  pretest  than  all 
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other  subgroups,  including  those  who  had 
attended  general  management  courses  and 
those  who  had  attended  no  training  at  all. 

In  addition  to  these  primary  effects,  the 
interactions  among  factors  provided  sev¬ 
eral  interesting  insights.  For  example,  the 
results  indicated  that  technical  manage¬ 
ment  experi¬ 
ence  can  go  far 
to  overcome 
any  disadvan¬ 
tages  associ¬ 
ated  with  non¬ 
technical  for¬ 
mal  education. 
The  relative 
advantages  (in 
terms  of  pretest  score)  associated  with 
technical  education  are  largely  isolated  to 
those  who  have  little  or  no  experience. 
When  both  had  experience  in  program 
management  offices,  technically  educated 
and  nontechnically  educated  students 
essentially  performed  equally. 

In  this  sense,  education  can  be  viewed 
as  a  means  of  achieving  the  equivalent  of 
a  level  of  experience  prior  to  actually 
entering  the  technical  management  envi¬ 
ronment.  Similarly,  the  study  did  not  in¬ 
dicate  that  the  time  interval  elapsed  since 
training  was  influential  in  determining 
pretest  scores.  When  the  interaction  of  the 
elapsed  interval  and  experience  was  evalu¬ 
ated,  the  results  indicated  that  experience 
more  than  accounted  for  any  losses  due 
to  the  passage  of  time  since  training. 

Discussion  and  Conclusions 


These  data  support  the  hypothesis  that 
students  react  more  favorably  to  instruc¬ 
tion  that  uses  more  experiential,  active 


approaches  to  learning.  While  this  and 
more  traditional  approaches  to  teaching 
result  in  positive  learning  experiences  for 
the  student,  the  work  documented  here 
further  indicates  that  students  learn  more 
when  experiential  teaching  methods  are 
used  in  training.  These  findings  in  com¬ 
bination  should  encourage  designers  of 
technical  training  to  structure  courses  that 
feature  hands-on  learning  where  theoreti¬ 
cal  and  conceptual  topics  are  reinforced 
by  experiences  gained  during  the  training. 
Too  many  courses  relegate  applications 
and  lessons  learned  through  experience  to 
the  post-training  period. 

This  study  found  the  differences  in 
learning  produced  by  the  two  methods 
investigated  to  be  smaller  than  expected; 
other  studies  have  found  no  conclusive 
differences  at  all.  There  is  a  possibility 
that  the  problem  has  been  in  the  instru¬ 
ments  used  to  measure  learning.  When  an 
identical  instrument  is  used  to  measure 
knowledge  levels  between  groups  (as  was 
the  case  in  this  study  where  the  same 
pretests  and  posttests  were  used  for  each 
of  the  two  groups),  comparisons  are  natu¬ 
rally  restricted  to  the  knowledge  levels 
measured  by  the  instrument.  In  the  case 
of  most  objective  tests,  knowledge 
measured  is  limited  to  levels  two  or  three 
of  the  Bloom  taxonomy  (1956). 

It  is  possible  that  the  instruments  used 
in  this  study  did  not  capture  the  full  extent 
of  learning  in  the  experiential  course  of 
training.  For  example,  students  synthe¬ 
sized  designs,  evaluated  them,  fabricated 
models  based  on  the  designs,  and  then 
evaluated  the  extent  to  which  the  products 
met  original  requirements.  The  tests  used, 
however,  did  not  address  the  higher  levels 
of  learning  associated  with  this  activity. 
This  study  was  able  to  demonstrate 


"This  study  found 
the  differences  i n 
learning  produced 
by  the  two  methods 
investigated  to 
be  smaller  than 
expected...." 
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statistical  differences  between  methods, 
due  in  some  part  to  the  large  size  of  the 
control  and  treatment  groups  involved 
(nominally  360  and  420). 

Future  studies  should  take  care  to  make 
certain  that  measurements  of  results  fully 
account  for  all  of  the  learning  involved. 
If  they  do,  more  conclusive  support  for 
the  use  of  experiential  methods  may  be 
found.  Additional  research  in  this  area 
would  be  useful,  since  the  current  litera¬ 
ture  is  largely  inconclusive  on  the  topic 
of  differences  in  learning  outcomes  as  a 
function  of  instructional  methods  used. 

This  research  also  supports  earlier  work 
that  found  little  or  no  relationship  between 
student  reactions  to  training  and  the 
learning  that  students  achieve.  Few  orga¬ 
nizations  evaluate  training  at  all;  and  many 
courses  evaluate  training  only  at  the  level 
of  student  reactions,  assuming  that  posi¬ 
tive  reactions  will  result  in  positive 
learning  and  transfer  of  training  to  the 
workplace.  That  is  apparently  not  a  well- 
justified  assumption.  The  evidence  is 
conclusive  that  training  evaluation  must 
include  an  assessment  of  learning  to  be 
of  value  as  an  indicator  of  training 
effectiveness.  Training  is  too  costly  and 
too  important  to  the  future  of  both 
organizations  and  their  people  to  tolerate 
less. 

Finally,  this  study  addressed  the  extent 
to  which  various  factors  influence  tech¬ 
nical  management  knowledge  levels  in 
general.  Education  type  (technical  or 
otherwise),  previous  management  expe¬ 
rience,  and  the  nature  of  previous  training 
appear  to  be  most  influential.  Surpris¬ 
ingly,  neither  the  interval  since  most 
recent  training  nor  the  nature  of  the  job 
currently  held  appeared  to  have  substan¬ 
tial  impact  on  knowledge  levels.  This  has 


a  number  of  interesting  implications.  For 
example,  engineers  generally  need  addi¬ 
tional  education  or  management  experi¬ 
ence  to  be  good  technical  managers;  they 
do  not  acquire  technical  management 
knowledge  naturally  as  a  by-product  of 
their  technical  expertise.  Another  obser¬ 
vation  was  that  training,  to  be  most  effec¬ 
tive,  needs  to  be  functionally  focused. 
Courses  that  address  too  broad  an  array 
of  functional  topics  will  likely  prove 
ineffective  at  training  in  any  of  them. 

These  findings  must  be  balanced 
against  certain  limitations,  including  those 
associated  with  the  instruments  used  to 
evaluate  the  learning  achieved  through  the 
use  of  experiential  methods.  It  is  possible 
that  the  results  documented  did  not 
account  for  all  the  learning  achieved. 

Another  consideration  is  that  the 
domain  knowledge  investigated  is  specific 
to  the  engineering  management  models 
that  are  common  to  the  Department  of 
Defense  and  to 
much  of  the 
U.S.  aerospace 
industry  as  re¬ 
flected  in  cur¬ 
rent  industry 
standards,  such 
as  the  Elec¬ 
tronic  Indus¬ 
tries  Associa¬ 
tion  (EIA,  1994)  and  Institute  of  Electri¬ 
cal  and  Electronics  Engineers  (IEEE, 
1994)  standards  for  systems  engineering 
management.  But  the  findings  may  not 
be  applicable  to  nontechnical  fields  or  to 
environments  in  which  the  standard 
models  for  engineering  management  do 
not  apply. 

In  spite  of  these  limitations,  this  work 
extends  previous  analytic  work  and 


"Another 
observation  was 
that  training,  to 
be  most  effective, 
needs  to  be 
functional  ly 
focused." 
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confirms  theoretical  work  in  an  area  where 
empirical  results  are  sparse.  It  also  illu¬ 
minates  several  areas — associated  with 


Paining  individuals  to  perform  as  engi¬ 
neering  managers — that  merit  further 
study. 
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